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Most  current  A1  technology  has  been  baaed  on  pro^sitionally  represented  theoretical  knowledge. 

that  if  AI  ia  to  accomplish  its  goals,  especially  in  the  tasks  of  sensory  interpretaticm  and  sensorimotor 
coordination,  then  it  must  solve  the  problem  of  representing  jmbodied  praetieal  knowledge.  Biological  evi¬ 
dence  shows  that  animals  use  this  knowledge  in  a  way  very  different  from  digital  computation.  This  sug¬ 
gests  that  if  these  problems  are  to  be  siplved,  then  we  will  need  a  new  breed  of  compujters,  which  we  call 
field  computers.  Examples  of  field  computers  are:  neurocomputers,  optical  computers,  imolecular  comput¬ 
ers,  and  any  kind  of  massively  parallel  ^nalog  computer.  We  claim5^that  the  principle  characteristic  of  ail 
these  computers  ia  their  massive  parailelum,  but  we  use  this  term  in  a  special  way.  We  argue^that  true 
massive  parallelism  comes  when  the  number  of  processors  ia  so  large  that  it  can  be  considered  a  continuous 
quantity.  Designing  and  programming  these  computers  requires  a  new  theory  of  computation,  one  version 
of  which  ia  presented  in  this  paper.  Wa  describe  a  univertal  field  computer,  that  is,  a  field  computer  that 
can  emulate  any  other  field  computer.  It  is  based  on  a  generalization  of  Taylor’s  theorem  to  continuous 
dimensional  vector  spaces  A  number  of  field  computations  are  illustrated,  including  several  transforma¬ 
tions  useful  in  image  understanding,  and  a  continuous  version  of  Kosko’s  bidirectional  associative  memory. 
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1.  AN  EXTENDED  THEORY  OF  KNOWLEDGE 

“Thon^  it  be  allowed,  that  reaeon  man  form  very  plaueible  eonjeetures  with  regard  to  eoneequeneee  of  each 
a  partietdar  conduct  in  each  particular  eireumetaneee;  it  it  still  supposed  imperfect,  without  assistance  of 
experience,  which  is  alone  able  to  give  stability  and  certainty  to  the  maxims,  derived  from  study  and 
reflection.  ” 

—  David  Hume 

I.l  The  “New”  AI 

We  argue  that  AI  is  moving  into  a  new  phase  characterised  by  a  broadened  understanding  of  the  nature  of 
knowledge,  and  by  the  use  of  new  computational  paradigms.  A  sign  of  this  transition  is  the  growing 
interest  in  neurocomputers,  optical  computers,  molecular  computers  and  a  new  generation  of  massively 
parallel  analog  computers.  In  this  section  we  outline  the  forces  driving  the  development  of  this  “new”  AI. 
In  the  remainder  of  the  paper  we  present  the  theory  of  field  computers,  which  is  intended  to  be  a 
comprehensive  5'amework  for  this  new  paradigm. 

The  “old”  AI  has  been  quite  successful  in  performing  a  number  of  difficult  tasks,  such  as  theorem  prov* 
ing,  chess  playing,  medical  diagnosis  and  oil  exploration.  These  are  tasks  that  have  traditionally  required 
human  intelligence  and  considerable  specialised  knowledge.  On  the  other  hand,  there  is  another  class  of 
tasks  in  which  the  old  AI  has  made  slower  progress,  such  as  speech  understanding,  image  understanding, 
and  sensorimotor  coordination.  It  is  interesting  that  these  tasks  apparently  require  less  intelligence  and 
knowledge  than  do  the  tasks  that  have  been  successfully  attacked.  Indeed,  most  of  these  recalcitrant  tasks 
are  performed  skillfully  by  animals  endowed  with  much  simpler  nervous  systems  than  our  own.  How  is 
this  possible? 

It  is  apparent  that  animals  perform  (at  least  some)  cognitive  tasks  very  differently  from  computers. 
Neurons  are  slow  devices.  The  well-known  “Hundred  Step  Rule”*  says  that  there  cannot  be  more  than 
about  a  hundred  sequential  processing  steps  between  sensory  input  and  motor  output.  This  suggests  that 
nervous  systems  perform  sensorimotor  tasks  by  relatively  shallow,  but  very  wide  (i.e.,  massively  parallel) 
proccaaing  Traditional  AI  technology  depends  on  the  digital  computer’s  ability  to  do  very  deep  (millions 
of  sequential  operations),  but  narrow  (1  to  100  processors)  p  ocessing.  Neurocomputing  is  an  attempt  to 
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obtain  some  of  the  advantages  of  the  way  animals  do  things  by  direct  emulation  of  their  nervous  systems. 
1.3  Theoretical  and  Practical  Knowledge 

One  visible  difference  between  the  old  and  new  AIs  is  in  their  computational  strategies;  the  former  stresses 
deep  but  narrow  processing,  that  latter  shallow  but  wide  processing.  Underlying  this  difference  .nere  is  a 
deeper  one:  a  difference  in  theories  of  knowledge.  The  old  AI  emphasizes  propositional  (verbaiizable) 
knowledge.  That  is,  it  assumes  that  all  knowledge  can  be  represented  by  sentence-iike  constructs  (i.e., 
finite  ensembles  of  discrete  symbols  arranged  in  accord  with  definite  syntactic  rules).  The  propositional 
view  is  not  new;  it  goes  very  far  back,  arguably  to  Pythagoras.  Yet  there  is  considerable  evidence  that 
nonpropositional  knowledge  is  at  least  as  importtmt.^ 

The  problems  of  practical  action,  as  opposed  to  theoretical  contemplation,  are  too  complicated  for  pro- 
positional  analysis.  The  real  world  is  simply  too  messy  for  idealized  theories  to  work.  Representation  in 
terms  of  discrete  categories,  and  cognition  by  manipulation  of  discrete  structures  referring  to  these 
categories,  may  be  appropriate  to  the  idealized  worlds  of  chess  playing  and  theorem  proving  (although  even 
this  is  doubtful^).  However,  in  practical  action  the  context  looms  large,  as  does  the  indefiniteness  of 
categories  and  the  other  second  order  effects  that  propositional  representation  routinely  idealizes  away. 

Of  course,  the  approximations  of  propositional  representation  can  be  improved  by  a  deeper  theoretical 
analysis,  but  this  greatly  increases  the  computational  burden.  Traditional  AI  is  faced  with  a  dilemma: 
simple  theories  do  not  enable  skillful  behavior,  but  detailed  theories  are  computationally  infeasible.  There 
might  seem  to  be  no  way  to  avoid  this  tradeoff.  But,  recalling  the  Hundred  Step  Rule,  and  observing  that 
animals  behave  skillfully,  we  realize  that  there  must  be  a  third  alternative. 

The  limitations  of  traditional  AI  technology  show  us  the  limitations  of  theoretical  knowledge,  i.e.. 
knowledge  that.  There  is,  however,  another  kind  of  knowledge,  which  we  can  call  practical  knowledge,  or 
knowledge  how.  For  example,  a  fish  knows  how  to  maintain  its  depth  in  the  water,  but  it  does  not  know 
that  neutral  buoyancy  is  achieved  by  adjusting  its  specific  gravity  to  that  of  water.  The  fish  does  not  have 
an  explicit  (propositional)  theory  of  how  temperature,  dissolved  substances,  etc.  affect  the  specific  gravity 
of  water,  nor  does  it  know  equations  describing  the  complex  manner  in  which  its  specific  gravity  depends 
on  the  state  of  its  body  (food  in  gullet,  air  in  air  bladder,  etc.  etc.).  Rather,  the  fish’s  knowledge  how  is 


represented  nonpropositionally  in  its  nervous  system  and  body. 

1.S  The  Acquisition  of  Knowledge 

The  foregoing  suggests  that  by  discovering  how  to  represent  and  manipulate  practical  knowledge  the  new 
'AI  may  accomplish  what  the  old  could  not.  There  are  difficulties,  however.  How  is  practical  knowledge 
acquired?  There  are  several  ways  theoretical  knowledge  is  acquired:  for  example,  it  may  be  taught.  Since 
propositions  can  be  encoded  in  verbal  structures,  language  can  be  used  to  transfer  theoretical  knowledge 
from  one  person  to  another.  (Of  course,  more  detailed  theories  require  correspondingly  larger  verbal  struc¬ 
tures  for  their  encoding.)  Thus,  in  principle,  the  representation  of  theoretical  knowledge  in  a  computer  is 
straight  forward;  we  merely  have  to  design  an  appropriate  knowledge  representation  language.  In  effect 
theoretical  knowledge  is  transferred  from  human  to  computer  in  the  same  way  it  is  transferred  from  human 
to  human. 

Before  theoretical  knowledge  can  be  transferred  it  must  be  acquired  in  the  first  place.  The  original 
discovery  of  theoretical  knowledge  is  beyond  the  scope  of  this  paper.  Here  we  restrict  ourselves  to  the 
transfer  of  theoretical  knowledge  from  one  person  to  another;  this  is  the  case  that  is  most  important  for 
expert  systems  and  other  applications  of  traditional  AI  technology. 

Since  practical  knowledge  is  nonpropositional,  it  cannot  be  encoded  verbally.  This  does  not  mean  it 
cannot  be  taught,  however,  since  we  can  teach  by  thawing  as  well  as  by  toying.  Therefore,  although 
.theoretical  knowledge  is  transferred  by  telling,  practical  knowledge  is  transferred  is  by  training.  Indeed  we 
often  speak  of  “training”  a  neural  network  to  accomplish  some  task. 

We  have  seen  how  practical  knowledge  may  be  transferred.  How  is  it  acquired  in  the  first  place?  In  a 
word,  by  ^idaptation.  In  nature  adaptation  occurs  predominantly  at  two  levels:  at  the  species  level  it  leads 
to  innate  practical  knowledge:  at  the  individual  level  it  leads  to  learned  practical  knowledge.  The  forego¬ 
ing  suggests  that  where  the  old  .AI  depended  on  verbal  encoding  and  transfer,  the  new  .AI  will  emphasize 
training  and  adaptation  as  means  of  knowledge  acquisition. 


a.  FIELD  TRANSFORMATION  COMPUTERS 


1.1  Maaaive  Parallelicm 

The  preceding  section  suggests  that  the  new  AI  will  augment  the  traditional  deep,  narrow  computation 
with  shallow,  wide  computation.  That  is,  the  new  AI  will  exploit  mastive  paralUium.  Now,  massive 
parallelism  means  different  things  to  different  people;  massive  parallelism  may  begin  with  a  hundred,  a 
thousand,  or  a  million  processors.  On  the  other  hand,  biological  evidence  suggests  that  skillful  behavior 
requires  a  very  large  number  of  processors,  so  many  in  fact  that  it  is  infeasible  to  treat  them  individually; 
they  must  be  treated  en  matte.  This  has  motivated  us  to  propose^  the  following  definition  of  massive 
parallelism: 

Definition  (Massive  Parallelism):  computational  system  is  mattively  parallel  if  the  number  of  pro¬ 

cessing  elements  is  so  large  that  it  may  conveniently  be  considered  a  continuous  quantity. 

That  is,  a  system  is  mattively  parallel  if  the  processing  elements  can  be  considered  a  eontinuout  matt 
rather  than  a  diterete  entemble. 

How  large  a  number  is  large  enough  to  be  considered  a  continuous  quantity?  That  depends  on  the  pur¬ 
pose  at  hand.  A  hundred  is  probably  never  large  enough;  a  million  is  probably  always  large  enough;  a 
thousand  or  ten  thousand  may  be  enough.  One  of  the  determining  factors  will  be  whether  the  number  is 
large  enough  to  permit  the  application  of  continuous  mathematics,  which  is  generally  more  tractable  than 
discrete  mathematics. 

We  propose  this  definition  of  massive  parallelism  for  a  number  of  reasons.  First,  as  noted  above,  skill¬ 
ful  behavior  seems  to  require  significant  neural  mass.  Second,  we  are  interested  in  computers,  such  as  opti¬ 
cal  computers  and  molecular  computers,  for  which  the  number  of  processing  elements  is  effectively  continu¬ 
ous.  Third,  continuous  mathematics  is  generally  easier  than  discrete  mathematics.  And  fourth,  we  want 
to  encourage  a  new  style  of  thinking  about  parallelism.  Currently,  we  try  to  apply  to  parallel  machines  the 
thought  habits  we  have  acquired  from  thinking  about  sequential  machines.  This  strategy  works  fairly  well 
when  the  degree  of  parallelism  is  low,  but  it  will  not  scale  up.  One  cannot  think  individually  about  the 
102°  processors  of  a  molecular  computer.  Rather  than  postpone  the  inevitable,  we  think  that  we  should 
begin  now  to  develop  a  theoretical  framework  for  understanding  massively  parallel  computers.  The 
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principal  goal  of  this  paper  is  to  propose  such  a  theory. 

3.3  Field  Transformation 

Our  aim  then  is  to  develop  a  way  of  looking  at  massive  parallelism  that  encompasses  a  variety  of  imple¬ 
mentation  technologies,  including  neural  networks,  optical  computers,  molecular  computers  and  a  new  gen¬ 
eration  of  analog  computers.  What  these  all  have  in  common  is  the  ability  to  process  in  parallel  amounts 
of  data  so  massive  as  to  be  considered  a  continuous  quantity.  This  suggests  that  we  structure  our  theory 
around  the  idea  of  a  field,  i.e.  a  continuous  (dense)  ensemble  of  data.  We  have  in  mind  both  scalar  fields 
(such  as  potential  fields)  and  vector  fields  (such  as  gravitational  fields).  Any  operation  on  such  a  field, 
either  to  produce  another  field  or  to  produce  a  new  state  of  the  field,  can  be  considered  massively  parallel, 
since  it  operates  on  all  the  elements  of  the  field  in  parallel.  Indeed,  it  would  not  be  feasible  to  serialize  the 
processing  of  the  field;  modest  degrees  of  parallelism  cannot  cope  with  the  large  number  of  field  elements. 

In  the  remainder  of  this  paper  we  explore  field  tramformation  computers,  that  is,  computers  character¬ 
ized  by  the  ability  to  perform  (in  parallel)  transformations  on  scalar  and  vector  fields.  We  are  not  suggest¬ 
ing  that  field  computers  are  unable  to  perform  scalar  calculations;  in  fact  we  assume  that  field  transforma¬ 
tion  computers  have  the  scalar  capabilities  of  conventional  digital  and  analog  computers.  Scalars  have 
many  uses  in  field  computation.  For  example,  we  may  want  to  use  a  scalar  parameter  to  control  the  rate 
at  which  a  field  transformation  takes  place  (e.g.,  a  reaction  rate  in  a  molecular  computer).  Similarly,  we 
may  use  a  scalar  representing  the  average  intensity  of  a  field  to  control  the  contrast  enhancement  of  that 
field.  A  scalar  threshold  value  may  be  used  to  suppress  low  level  noise,  and  so  forth. 

An  important  reason  for  combining  field  computation  with  conventional  digital  computation  is  that  it 

permits  knowing  how  to  be  combined  with  knowing  that,  leading  to  knowledgeable,  skillful  behavior.  The 

»> 

combined  use  of  propositional  and  theoretical  knowledge  is  unfortunately  beyond  the  scope  of  this  paper.* 

3.3  Classes  of  Field  Transformations 

Field  transformations,  like  filters,  can  be  divided  into  two  classes:  nonrecur«ti/«  and  recursive.  A  nonre- 
cursive  transformation  is  simply  a  functional  composition  of  more  elementary  transformations.  The  output 
of  a  nonrecursive  transformation  depends  only  on  its  input.  A  recursive  transformation  involves  some  kind 
of  feedback.  Hence,  its  output  depends  both  on  its  input  and  on  its  prior  state.  Recursive  transformations 
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we  ideal  for  simulating  the  temporal  behavior  of  a  physical  system,  for  example  in  simulated  annealing* 
and  Boltsmann  machines.* 

Z.a  General  Purpose  Field  Computers 

Many  field  computers  are  designed  for  special  purposes;  this  has  been  the  case  with  field  computers  to  date, 
and  we  expect  it  to  be  the  case  in  the  future.  In  these  computers,  devices  implementing  field  transforma¬ 
tions  (such  as  filters  and  convolutions)  are  assembled  to  solve  a  small  class  of  problems  (e.g.,  pattern  recog¬ 
nition).  On  the  other  hand,  our  experience  with  digital  computation  has  shown  us  the  value  of  general 
purpose  or  programmable  computers.  This  architectural  feature  permits  one  computer  to  perform  a  variety 
of  digital  computations,  which  eliminates  the  need  to  construct  special  purpose  devices,  and  speeds  implcr 
mentation  of  digital  algorithms. 

The  foregoing  observations  suggest  that  general  purpose  field  computers  will  be  similarly  valuable.  In 
these  the  connections  between  field  transformation  units  and  field  storage  units  are  programmable,  thus 
fwilitating  their  reconnection  for  a  variety  of  purposes.  In  fact,  we  may  want  to  make  better  use  of  our 
resources  by  multiplexing  the  use  of  field  transformation  units  under  the  control  of  a  program.  Thus,  a  pro¬ 
gram  for  a  general  purpose  field  computer  might  look  very  much  like  a  conventional  program,  except  that 
the  basic  operations  are  field  transformations  rather  than  scalar  withmetic. 

We  cannot  build  into  a  general  purpose  field  computer  every  transformation  we  might  need.  Instead  we 
must  choose  a  set  of  primitive  operations  that  permit  the  programming  of  all  others.  How  can  such  a  set 
of  primitive  operations  be  chosen?  How  can  we  be  guaranteed  that  we  have  provided  all  the  necessary 
facilities?  For  digital  computers  this  question  is  answered  in  part  by  computability  theory.  For  example, 
this  theory  shows  us  how  to  construct  a  nntversal  Turing  machine,  which,  given  an  appropriate  program, 
can  emulate  any  Turing  machine,  .\lthough  the  universal  Turing  machine  is  hardly  a  practical  general 
purpose  computer,  consideration  of  it  and  other  universal  machines  shows  us  the  kinds  of  facilities  a  com¬ 
puter  must  have  in  order  to  be  universal.  There  follows  the  hard  engineering  job  of  going  from  the  theoret¬ 
ically  sufficient  architecture  to  the  practically  necessary  architecture. 

Can  the  same  be  accomplished  for  field  computers?  Is  there  a  universal  field  computer  that  can  emulate 
any  field  computer?  If  there  is  such  a  thing,  then  we  can  expect  that  it  may  form  a  basis  for  practical  gen- 


enl  purpose  field  computers  in  much  the  same  way  that  Turing  machines  do  for  digital  computers.  In  the 
next  section  we  prove  that  general  purpose  field  computation  is  possible. 

».  A  UNIVERSAL  FIELD  COMPUTER 

5.1  Introduction 

In  this  section  we  develop  the  general  theory  of  field  computation  and  prove  the  existence  of  a  universal 
field  computer.  In  particular,  we  show  that  with  a  certain  sec  of  built  in  field  transformations  we  can 
implement  (to  a  desired  degree  of  accuracy)  any  field  transformation  in  a  very  wide  class.  This  is  analo¬ 
gous  to  the  result  from  Turing  machine  theory:  The  universal  Turing  machine  allows  us  to  implement  (to 
a  desired  degree  of  accuracy)  any  function  in  a  wide  class  (now  known  as  the  computable  functiont). 

The  phrase  to  a  desired  degree  of  accuracy’  appears  in  both  of  the  preceding  statements.  What  does  it 
mean’  For  the  Turing  machine  it  means  that  a  given  accuracy  (e.g.,  precision  or  range  of  argument)  can 
be  achieved  by  providing  a  long  enough  tape.  For  the  digital  computer  it  means  that  computations  are 
normally  performed  to  a  given  precision  (e.g.,  the  word  length),  and  that  finite  increments  in  the  desired 
precision  require  finite  increments  in  the  resources  required  (e.g.,  additional  registers  and  memory  cells  for 
double  and  multiple  precision  results,  or  stack  space  for  recursion).  The  case  is  much  the  same  for  the 
uaiversal  field  computer.  Finite  increments  in  the  desired  accuracy  of  a  field  transformation  will  require 
finite  increments  in  the  resources  used  (such  as  field  transformation  and  storage  units). 

There  are  a  number  of  theoretical  bases  for  a  universal  field  computer.  We  have  investigated  designs 
based  on  Fourier  analysis,  interpiolation  theory  and  Taylor’s  theorem,  all  generalized  for  field  transforma¬ 
tions.  In  this  paper  we  present  the  design  based  on  Taylor’s  theorem.  There  are  no  doubt  ais  many  princi- 
oles  upon  which  universal  field  computers  can  iie  based  .as  there  are  bases  for  universal  digital  computers. 

5.2  Taylor  Series  Approximation  of  Field  Transforms 

In  this  section  we  develop  the  basic  theory  of  functions  on  scalar  and  vector  fields  and  of  their  approxima¬ 
tion  by  Taylor  series.  Most  of  the  definitions  and  theorems  in  sections  3.2  and  3.3  have  been  previously 
pxMished^;  they  are  reproduced  here  for  completeness.  Once  it  is  understood  that  fields  are  treated  as 
continuous-dimensional  vectors,  it  will  seen  that  the  mathematics  is  essentially  that  of  finite-dimensional 
vectors.  Note  that  the  treatment  here  is  heuristic  rather  than  rigorous.  First  we  consider  scalar  fields; 


later  we  turn  to  vector  fields. 


t 

i 

I 


As  usual  we  take  a  iicalar  field  to  be  a  function  4>  from  an  underlying  set  0  to  an  algebraic  field  K,  thus 
<l>:  Q  -*  K.  For  our  purposes  K  will  be  the  field  of  real  numbers,  IR.  We  use  the  not.T.tion  for  the 

set  of  all  scalar  fields  over  the  underlying  set  fl  (/f  =  IR  being  understood).  Thus,  is  a  function 

space,  and  in  fact  a  linear  space  under  the  following  definitions  of  field  sum  and  scalar  product; 

{4>  -r  ^)t  =  <?l 

{u)t  =  A(^,)  (1) 

Note  that  we  often  write  <(>^  for  the  value  of  the  field  at  the  point  t.  As  a  basis  for  this  linear  space 

we  take  the  unit  functions  U;  for  each  1  €  fl.  They  are  defined 

=  1 

a;,(s)  =  0,  if  (2) 


Note  that 


4>  = 


I 


■*! 

i' 


The  preceding  definitions  show  that  we  can  think  of  scalar  fields  as  vectors  over  the  set  fl.  Since  we  want 
to  be  quite  general,  we  assume  only  that  H  is  a  measurable  space.  In  practice,  it  will  usually  be  a  closed 
and  bounded  subspace  of  £”*,  n-dimensional  Euclidean  space.  Thus  we  typically  have  one,  two  and  three 
dimensional  closed  and  bounded  scalar  fields. 

Since  0  is  a  measure  space,  we  can  define  an  inner  product  between  scalar  fields: 

0  ■  w  =  (Pixpi  dt. 


We  also  define  the  norm: 


dt. 


Thus  ^(n)  is  the  function  space  L^{Vl).  Note  that  the  w,  are  not  an  orthogonal  set  under  this  norm,  since 

|'a/,||  =  0, 


We  first  consider  scalar  valued  functions  of  scalar  fields,  that  is  functions  /:  ^(H)  -*  H.  We  prove 


some  basic  properties  of  these  functions,  culminating  in  Taylor’s  theorem. 


Definition  (Differentiability):  Suppose  /  is  a  scalar  valued  function  of  scalar  fields,  /:  $(0)  -*  R,  and 
that  /  is  defined  on  a  neighborhood  of  €  ^(H).  Then  we  say  that  /  is  difftrtntxablt  at  ^  if  there  is  a 
field  G  6  ^(n)  such  that  for  all  a  in  this  neighborhood 


^  ot)  -  f{4>)  =  + 'lilo'll 


(6) 


where  17  -»  0  as  l|a||  ->0.  We  wiU  later  call  G  the  gradient  of  /  at 


Tbcwem:  If  /  is  differentiable  at  ^  then  /  is  continuous  at 

Pro«f:  Since  /  is  differentiable  at  (f>  we  know 

fW  -  -  <i>yG  +  r}\\rp  -  <i>\\. 

Therefore, 

!  /{V’)  -  f{4)\  •  =  I  hi)  -  <t>yG  +  fjllv*  -  ^11  I 
^1(0-  <i>yG\  +  1  rj\  11^  -  ^i| 

^  ||G||  11^  -  ^11  +  I  >7|  110  -  <t>\\ 

=  (IlCIi  +  I'll)  !|0- 0(1. 

Thus  /  is  continuous  at  0.  ■ 

The  quantity  a  '  G  whose  existence  is  guaranteed  by  differentiability  is  called  the  directional  deriva¬ 
tive  of  /  with  respect  to  a  at  0.  It  is  defined  directly  as  follows. 

Definition  (Directional  Derivative):  The  directional  derivative  in  the  “direction”  a  is  given  by  the 
following  limit: 

V./(0)  ’  lin,  /(»  -  m 

da  A-0  h  7) 


We  use  Vo/  df/da  interchangeably  for  the  directional  derivative.  Note  that  if  /  is  differentiable  at  0 

then  Va/(0)  =  a  •  G. 
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Lemma:  If  /  is  differentiable  in  a  neighborhood  of  then 

f{<t>  +  za)  =*  Vo/(^  +  *«)• 


(8) 


Proof:  By  the  definition  of  the  derivative; 

±  H4,  -  zo,)  -  llm  /!■»  "  -  M  " 

dz  A-o  A 

_  J[4>  xa  +  ka)  —  /(^  +  za) 

k-o  k 

=  Va/(<P 

The  last  step  follows  by  the  definition  of  the  directional  derivative,  a 

Theorem  (Mean  Value):  Suppose  /:  ^(0)  IR  is  continuous  on  a  neighborhood  containing  (j)  and  tf>. 
Then,  there  isa^,  I,  such  that 

fW  -  =  (^  -  <i>)  •  V/(x) 

where  x  -  ^  ~ 

Proof:  Let  a  =  tp  —  <t>  and  consider  the  function 

F{x)  =  /(d>  za)  -  /(<!>)  -  zl/(0)  - 

Since  /  is  continuous,  so  is  f .  Now,  since  F(0)  =  F(l)  =  0,  we  have  by  Rolle’s  Theorem  that  there  is  a 
0  ^  ^  <  1,  such  that  F'  {ff)  =  0.  Note  that 

I 

F'{x)  =  ^  -  fW  -  -  fi9)\ 

=  ~  [/(^)  "  fi9)]  ■ 

By  the  preceding  lemma 

F'{z)  =  Vj{4>  +  xa)  -  [/(^)  -  f  (</>)] 
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Hokc,  substituting  $  for  x, 


0  .  r[e)  =  +  <?«)  -  \m  -  im- 

Therefc»u,  transposing  we  have 


f{i>)  -  f{4>)  =  Va/(<^  + 


and  the  theorem  is  proved.  ■ 


Theorem  (Taylor):  Suppose  that  /  and  all  its  directional  derivatives  through  order  n  +  1  are  continuous 
in  a  nnghborhood  of  4>.  Then  for  all  a  such  that  ^  +  a  is  in  that  neighborhood  there  isa^,  0^^^  1, 
such  that 


j[6  +  a)  =  m  +  Va/(^)  +  -  •  ■ 


(10) 


Proof:  By  the  Taylor  theorem  on  real  variables, 


H4>  -  <«)  •  fw  +  i  mt  <•  ~ 


i»+i 


(n+1)! 


J[<t>  +  9a)t 


s+l 


Observe  that  by  the  preceding  lemma 


dl" 


f{<t>  +  ta)  =  +  ««)• 


Therefore, 


-  to)  =  m  -  r  \vimr  ~  ^  ^ 

i  n!  (n+l)! 

Setting  1=1  gives  the  desired  result.  ■ 


The  extension  to  a  function  of  teveral  scalar  fields  is  routine. 

Since  our  “vectors”  are  continuous  dimensional,  partial  derivatives  are  with  respect  to  a  “coordinate” 
t  €  n  rather  than  with  respect  to  a  coordinate  variable.  To  define  partial  derivatives  it’s  convenient  to 
make  use  of  the  Dirac  delta  functions,  6^,  for  16  0: 


S,(t)  =  cx> 

Sf(s)  =  0,  for  4  ^  t 


(11) 


Of  coarse,  by  the  first  equation  above  we  mean  Sf(s)  =  lim  e~^  for  |  $—t\  <  e/2.  Note  the  following  pro- 

<-*o 

parties  of  the  delta  functions  (fields): 

m\  =  1 

Si  •  =  4i 

Si  •  <^dt  = 

Gtvea  the  delta  functions  the  partial  derivative  of  /  at  coordinate  t  is  simply  expressed: 

M.  = 


Vs/ 


Theorem:  If  /  is  differentiable  at  <(>  then  the  first  order  partial  derivatives  exist  at 
Procf:  First  observe  that  by  differentiability 

^  hSi)  -  m  _  m^hSi-G-^nWhSiW-m 
h  h 

=  •G  +  7ll<J,ll|A|/A 
'=  Si  ■  G  +  r,\h\/h 
=  Gi  +  Ti\  h\  / h 

Recalling  that  r?  — ►  0  as  A  —  0.  observe 


lim 

it-«0 


f{<f>  +  hSi)  -  f{S>) 


-  G, 


=  lim  i(7,  +n\h\/h  -  G,| 

*-o 


=  lim  ( ri\ 
k-*Q 


(12) 


(13) 


=  0 


Hence,  =  Gj,  where  G  is  the  field  whose  existence  is  guaranteed  by  differentiability.  Thus  the 


dSi 
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partial  derivative  exists.  ■ 


What  is  the  field  G  whose  points  are  the  partial  derivatives?  It  is  just  the  gradient  of  the  function. 


Definition  (Gradient):  The  gradient  of  /;  $(12)  IR  at  is  a  field  whose  value  at  a  point  t  is  the  par- 

g 

tial  derivative  at  that  point,  — —f{4>)- 

dS, 


(14) 


Since,  by  Eq.  3,  V/(^)  =  L  ^2>e  gradient  can  also  be  expressed  in  terms  of  the  basis  fane- 

j  n 

tions  and  the  partial  derivatives: 


V/($)  =  J-/(<i)  dt. 


(15) 


When  no  confusion  will  result,  we  use  the  following  operator  notations: 


V/  =  d* 

V  =  /„u;,d/85.dt 

a/d«,  =  •  V  =  Vf, 


(16) 


Finally,  since  V/(0)  =  the  field  guaranteed  by  differentiability,  and  Vq/(^)  =  ct  •  G,  we  know 

($)  =  Va/(<?^)  =  a  •  v/($) 


(17) 


or,  in  operator  form,  d/da  =  Va  =  <>  '  V- 

3.3  A  Universal  Field  Computer  Based  on  Taylor  Series  Approximation 

We  can  use  Taylor’s  Theorem  to  derive  approximations  of  quite  a  general  class  of  scalar  valued  functions 
of  scalar  fields.  Thus,  if  we  equip  our  universal  field  computer  with  the  hardware  necessary  to  compute 
Taylor  series  approximations,  then  we  will  be  able  to  compute  any  of  a  wide  class  of  functions  (namely, 
those  functions  whose  first  n  partial  derivatives  exist  and  are  continuous).  Therefore,  consider  the  general 
form  of  an  n>term  Taylor  series: 
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What  hardware  is  required?  Clearly  we  will  need  a  field  subtractor  for  computing  the  difference  field 
a  =  ^  We  will  also  need  a  scalar  multiplier  for  scaling  each  term  by  l/k\\  we  will  also  need  a  scalar 

adder  for  adding  the  terms  together.  The  harder  problem  is  to  find  a  way  to  compute  Va/(^o)  ^  vector 

a  that  depends  on  the  (unknown)  input  <t>.  The  trouble  is  that  the  as  and  the  Vs  are  interleaved,  as  can 
be  seen  here: 

V*/(fl>o)  =  ("  •  V)*/(4io) 

=  (a-v)*-Ma-v/(<>o)l 


=  /n  '  “ 


/{^Pq)  dfi  df;  •  '  •  df* 


We  want  to  separate  everything  ..hat  depends  on  a,  and  is  thus  variable,  from  everything  that  depends  on 
/(^q),  and  is  thus  fixed.  This  can  be  accomplished  (albeit,  with  extravagant  use  of  our  dimensional 
resources)  by  means  of  an  outer  product  operation.  Therefore  we  define  the  outer  product  of  two  scalar 
fields: 


(<?  A  «>),,,  = 

Note  that  if  i®.  w  6  <&(!!)  then  0  \  iv  k  <bfn^). 


(19) 


To  see  how  the  outer  product  allows  the  variable  and  fixed  parts  to  be  separated,  consider  first  the  case 


V.V(#o)  -  ^  /(*o)  <“  d* 

=  /nX,(«  A  o).,,  (V).(V),  /(^o)  d* 

=  /n/n^“  ^  ^ 

=  /ni(®  A  a),(v  A  v),di  /(^q) 

=  (a  A  a)  •  (V  A  V)  /(<^o) 

Now  we  can  see  how  the  general  case  goes.  First  we  define  the  ib-fold  outer  product: 

=  (ji 

^l*+l|  =  A  (20) 


Then, 


V*/(^)  =  Of'*’  •  v‘*'/(^) 


The  n-term  Taylor  series  then  becomes 


(21) 


m  =  E-7r(^- •^0)1*1  •v'‘'/(*o) 

*=•0  *• 


(22) 


Since  0o  is  fixed,  we  can  compute  each  V**(/(<?^o)  once,  when  the  field  computer  is  programmed.  Then,  for 
sny  given  input  4>  compute  (^  -  take  the  inner  product  of  this  with  v(*^/(4^o)-  Thus,  in 

addition  to  the  components  mentioned  above,  computing  the  Taylor  series  approximation  also  requires 
outer  and  inner  product  units  that  will  accommodate  spaces  up  to  those  in  ^(n“). 

We  consider  a  very  simple  example  of  Taylor  series  approximation.  Suppose  we  want  to  approximate 

defint  <!>,  which  computes  the  definite  integral  of  defint  <^  =  f  ds.  First  we  determine  its  partial 

j  n 

derivative  at  t  by  observing: 
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defint  +  kS,)  -  defint  ^ 

lim - .  — 

k->o  h 


lim 

k-o 


,^»  +  H(«)  d*  -  In^,  d* 
h 


11^  i.  +  >/„<,(.)  d.  -  d, 

k-Q  A 


=  limAi|^,|l/A  =  1 

k— 0 


Thus,  ——  defint  ^  =  1,  &nd  we  can  see  that 
Ofl( 


V  defint  ^  =  1, 


(2S) 


where  1  is  the  constant  1  function,  =  1.  This  leads  to  a  one  term  Taylor  series,  which  is  exact: 

defint  ^  ^  •  1 


Note  that  1  is  a  fixed  field  that  must  be  loaded  into  the  computer. 


(24) 


S.4  Transformations  on  Scalar  and  Vector  Fields 


The  previous  results  apply  to  tealar  valued  functions  of  scalar  fields.  These  kinds  of  functions  are  useful 
(e.g.,  to  compute  the  average  value  of  a  scalar  field),  but  they  do  not  exploit  the  full  parallelism  of  a  field 
computer.  Achieving  this  requires  the  use  of  functions  that  accept  a  (scalar  or  vector)  field  as  input,  and 
return  a  field  as  output.  We  briefly  sketch  the  theory  for  scalar  field  valued  functions  of  scalar  fields: 
transformations  on  vector  fields  are  an  easy  extension  of  this. 

By  a  scai-ir  field  valued  transformation  of  scalar  fields  we  mean  a  function  F:  ->  <^(no).  Such  a 

transformation  is  considered  a  familv  of  scalar  valued  functions  /,;  $(nj)  -*■  IR  for  each  t  ‘z  Ho;  these  are 
the  component  function*  of  F.  Note  that  F  can  be  expressed  in  terms  of  its  components: 

=  /  fM  w,  dt 

(25) 


More  briefly,  F  =  j  /jW,  df.  F  is  decomposed  into  its  components  by  6,  •  F(^)  = 

Next  we  turn  to  the  differentiability  of  field  transformations.  To  define  this  it  is  necessary  to  first 
define  an  analog  to  the  inner  product  for  fields  of  different  dimension.  Thus,  we  define  the  continuous- 


-17- 


dimensional  analogne  of  a  vector-matrix  product: 


(W),  - 

(o«),  -  /„*.*>  -  *■*! 

where  the  transpose  of  a  field  is  defined  With  this  notation  differentiability  can  be  defined. 

Definition  (Differentiability  of  Field  Transformations):  Suppose  F:  -»  ^(Ho)  is  a  field 

valued  function  of  scalar  fields  defined  on  a  neighborhood  of  We  say  that  F  is  differentiable  at 

4  provided  there  is  a  field  F  6  $(0^x02)  such  that  for  all  a  in  the  neighborhood 


F(^  Qt)  —  F(^)  =  aF  -h  H||aj 


where  H  €  and  UHlj  -*  0  as  Ijall  -»  0.  We  will  show  that  F  is  the  gradient  of  F  at  <i>. 


(27) 


Next  we  consider  the  directional  derivative  of  a  field  transformation.  For  a  scalar  function  /,  Va/(^) 
is  a  scalar  that  describes  how  much  changes  when  its  argument  is  perturbed  by  a  small  amount  in  the 
“direction”  a.  For  a  field  transformation  F,  should  be  a  field,  each  component  of  which  reflects 

how  much  the  corresponding  component  of  F(^)  changes  when  ^  moves  in  the  “direction”  a.  That  is, 
iVoF(^)],  =  Va  •  F(^).  Hence, 


(28) 


VaF(^)  =  W.  Vo  •  F(«>)  dt, 

or,  more  briefly,  VoF  =  L  '^a  '  F  dt.  The  directional  derivative  b  defined  directly  by  a  limit. 

jn, 


Definition  (Directional  Derivative  of  Field  Transformations):  If  F:  $(f2J  —  $(no)  is 
differentieble  at  d  then 


VaF(<li)  =  ^(<P)  =  lim 

aa  k-o  h 


(29) 


It  b  obvious  that  VaF(4i)  =  crF,  where  F  is  the  field  whose  existence  is  guaranteed  by  differentiability. 
Tkb  suggests  the  definition: 

Definition  (Gradient  of  Field  Transformation):  The  gradient  of  F  at  ^  is  the  field  whose  value  at  a 
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poiat  t  is  the  partial  derivative  of  F  at  that  point; 


|VF(*)|,  - 


(SO) 


Since  V4,F(^)  =  ^,r  =  F,,  it  follows  that  vf  (^)  =  fi  “  expected. 


Notice  that  the  gradient  is  of  higher  dimension  than  the  field  transformation.  That  is,  if 
F:  then  yF:  4^(n|)  -»  ^(nj  x  flo).  Higher  order  gradients  will  have  similarly  higher 

order: 


v*F:  ^(n,)  -  $(n,*  X  n,),  for  F:  -  ^(n^) 


(31) 


A  derivation  similar  to  that  for  scalar  field  functions  yields  the  Taylor  series  for  field  transformations: 

F(^)  «  S  -^  [(^  -  ^>o)  Vi*F(flio) 

k=0  *• 


(32) 


As  before,  outer  products  can  be  used  to  separate  the  variable  and  fixed  components. 

We  illustrate  the  Taylor  series  computation  for  a  simple  but  important  class  of  field  transformations, 
the  integral  operators.  For  example,  the  derivative  and  difference  transformations,  which  are  very  useful  in 
image  processing,  are  integral  operators,  as  will  be  shown  below. 


Deftnition  (Integral  Operator):  A  field  transformation  F:  -♦  *^(112)  is  an  integral  operator  if 

there  is  a  field  ^  f^l)i  called  its  kernel,  such  that 


F(<?i)  =  ^<t> 

(33) 

Recall  (♦<&),  = 

Since  F((p  +  ip)  =  +  ip)  =  ^<p  +  ^ip  =  F{<p)  F(ip),  it’s  clear  that  integral  operators  are 

linear.  Thus,  their  gradients  are  esjjecially  easy  to  compute: 


Theorem  (Gradient  of  Integral  Operator):  The  gradient  of  an  integral  operator  is  the  transpose  of 


its  kernel. 


Pro0f:  Suppose  F(^)  =  is  an  integral  operator.  Observe  F(^  +  a)  -  F(^)  =  'J'a  =  a’f^,  where 
Hence  VFW  =  ■ 


Since  the  gradient  of  a  constant  function  is  zero,  all  higher  order  gradients  vanish,  so  the  Taylor  series  for 
an  integral  operator  is  exact:  F(^)  = 

Next  we  show  that  the  derivative  and  difference  operations  on  scalar  fields  are  integral  operators  and  we 
compute  their  Taylor  series.  First  consider  the  finite  difference  transformation  defined  so  that 


-  4>t-yk  ~ 

for  a  given  h  >  0.  To  see  that  this  is  an  integral  operator  observe: 


(34) 


^<i>t  =  ^i+k  '  '  <f>  =  {^t+k  ~  ^t)  '  *i> 


(35) 


Define  the  field  ^  by  +*(**)  “  follows  that  since 

(*0),  =  f  =  r  (.5<+*(«)  -  <5,(b)1<^>,  dtt  =  (S,^^  -  8t)  '  <t> 

•'"i  •'"i  (36) 

The  only  trouble  with  this  formula  for  the  finite  difference  is  that  the  field  ^  is  not  physically  realizable, 
since  it  makes  use  of  the  Dirac  functions.  In  practice  we  have  to  replace  and  by  finite  approxima¬ 
tions,  but  the  resulting  approximate  difference  transformation  is  still  an  integral  operator.  The  same 
applies  to  the  derivative  transformation,  which  can  be  approximated  by  the  approximations  to  the  first 
and  higher  order  derivatives. 


To  further  illustrate  the  Taylor  series  for  scalar  field  valued  transformations,  we  consider  pointwite 
transformations.  .\  pointwise  transformation  applies  a  scalar  function  (but  not  necessarily  the  same  func¬ 
tion)  to  every  element  of  a  field.  That  is,  F:  ^(11)  -*  ^(H)  is  a  pointwise  transformation  if  for  some 

/,:  R  -  K. 


1F('^)I,  -  !M,) 


(ST| 


Note  that 


Srn<P)  = 
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Lemnia:  If  F:  $(0)  $(0)  ia  a  pointwise  transformation,  then 

■F(^)  =  •aF'(^) 


where  [F  '(<;?)],  =  /,  and  /{ '  is  the  derivative  of  /,. 


Proof:  By  the  definition  of  the  derivative: 


„  r  -C.  ^  V  -Fii^+ha)  -S,  ■¥{</>) 

Va  it  •  F(fl>)  =  hm - - - 

ft-O  ft 


lim 

A-O 


/,[(d>+Aa),i  -  fti4>i) 


(39) 


/«(<?, +/»a,)  -  It{4>t) 

hm - : - 

A^o  h 


Ylm  aJ^'{<j>^)  +€j  Aa<|/A 

A-O 


=  <>tfi'{4>t) 


=  (5,  •  aF'(^) 


Theorem  (Directional  Derivative  of  Pointwise  Transformation):  If  F:  *^(0)  —  *^(11)  is  a  point- 
wise  transformation,  then 


VaF(<6)  =  oF'{0) 


(40) 


Proof:  Applying  the  preceding  lemma; 


VaF(^)  =  f^U,,  Va  •  F(^)  dt 
=  /pW,  S,  •  aF'(^)  di 


"  “ /n"^' 

=  aF'(^) 

The  last  step  follows  by  Eq.  12.  ■ 

Corollary:  The  directional  derivative  of  a  pointwise  transformation  is  a  pointwise  transformation. 

Proof:  This  follows  immediately  from  the  preceding  theorem.  ■ 

The  theorem  and  its  corollary  lead  to  the  Taylor  series  expansion  of  a  pointwise  transformation: 

Theorem  (Taylor  Series  for  Pointwise  Transformation):  If  F;  ^(fl)  $(0)  is  a  pointwise 

transformation,  and  its  component  functions  and  ail  their  derivatives  through  n  — 1  are  continuous  in  a 
neighborhood  of  <p,  then  for  all  a  such  that  <^-ta  is  in  that  neighborhood  there  is  a  field  0  ^  ^  1, 


such  that 

F(<p^a)  = 

*=o  ('*^1]' 

Here  F'**  is  the  i:th  derivative  of  F: 

p'‘V)i,  -  /,'*'(»,)  -  ! 

dz  ’,1=*, 

where  /,  =  (J.  •  F. 

Proof:  By  the  Taylor  theorem  for  functions  of  reals: 
iF(0  -  a);,  =  ^  a,) 

•  ETr»,v,'‘'(0,)  +  T-^or' 

-  S-Tryp'*'!^)!,  +  t-ttf 

JksQ  *•  *)• 
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Tlu»  results  permits  converting  T&ylor  series  for  sc&lar  functions  to  the  Taylor  series  for  the  corresponding 
pointwise  field  transformations.  Notice  also  that  the  resulting  series  do  not  require  outer  products,  pro¬ 
vided  we  have  a  field  multiplier,  =  <Pt^f 

We  illustrate  this  theorem  by  determining  a  Taylor  series  approximation  for  In,  the  function  that  com¬ 
putes  the  natural  logarithm  of  each  field  element. 


Theorem  (Taylor  Series  for  Pointwise  Logarithm):  Suppose  (In  (p),  =  In  Then 


In  =  (<p  -  1)  -  \{4>  -  1)^  +  -  1)^  - 


(43) 


provided  -  Ij  ^1  and  <pf  ^  0,  for  all  t  6  O. 

Proof:  Note  that  for  k>0,  In^*^  <f>  =  (  — 1)*“^(1:  -  1)!  /  Therefore,  for  1:>1, 

In'*  1  =  (-1)*~^(A:  -  l)!.  By  the  Taylor  theorem, 

iB  (1  +  a)  =  In  1  +  y;-i.a*lnl*)  1  ^  +  Ba) 

(n+1)! 


=  -  l)!  +  ^  0a) 

*=t  *• 


1 


*=i 


To  prove  the  theorem  let  a  =  <J>  —  1.  ■ 

We  consider  vector  fields  briefly.  Recall  that  any  three-dimensional  vector  field  $  can  be  considered 
three  scalar  fields  <p,  tp,  x  where 


+  X(k 


(44) 


Siinilariy,  a  function  that  returns  a  three-dimensional  vector  field  can  be  broken  down  into  three  functions 


thnt  return  scalar  fields.  Thus,  we  see  that  a  transformation  on  finite  dimensional  vector  fields  can  be 


implemented  by  &  finite  number  of  transformations  on  scalar  fields. 


To'  ensure  the  continuity  of  field  valued  functions,  certain  restrictions  must  be  placed  on  the  fields  per¬ 
mitted  as  arguments.  Although  these  restrictions  are  still  under  investigation,  we  believe  that  it  is 
sufficient  that  the  input  field’s  gradient  be  bounded  at  each  stage.  This  will  be  the  case  for  all  physically 
realisable  fields.  This  restriction  on  allowable  inputs  finds  its  analogy  in  digital  computers:  legal  input 
numbws  are  restricted  to  some  range;  numbers  outside  that  range  may  cause  underflow  or  overflow  in  the 
subsequent  computation.  In  the  same  way  here,  fields  whose  gradients  are  too  large  may  lead  to  incorrect 
results. 

4.  EXAMPLE  APPLICATION:  BIDIRECTIONAL  ASSOCIATIVE  FIELD  MEMORY 

In  this  section  we  illustrate  the  theory  of  field  computation  by  analyzing  a  continuous  version  of  Kosko’s 
bidirectional  associative  memory.^  The  system  operates  as  follows  (see  Figure  1). 


The  goal  is  to  store  a  number  of  associative  pairs  where  G  and  €  ^(no). 

(We  assume  these  fields  are  bipolar,  that  is,  take  values  in  {  —  1,  +1),  although  physical  realizability  actu¬ 
ally  re«{ures  continuous  variation  between  —1  and  -1-1.)  Presentation  of  a  field  <!>  €  at  ioj  eventu¬ 

ally  yields  at  outj  and  out2  the  pair  j  for  which  is  the  closest  match  for  (p.  Similarly,  presentation  of 
Ip  at  in^  will  yield  the  pair  for  which  is  the  closest  match.  The  pairs  are  stored  in  a  distributed 


fashion  in  the  field  4*  E  x  computed  as  follows: 

4  =  ^(i)  /\  ^(1)  +  . .  •  +  0(«)  A 

Note  that  A  0^*'  reflects  the  cross  correlations  between  the  values  of  and  the  values  of 


(45) 


Two  of  the  boxes  Fig.  1  perform  “matrix-vector”  multiplications,  ^<i>  or  ib'il.  Thus  presentation  of  a 
field  <f>  at  in^  yields 


tb'  -  5(^,  'td)) 


(46) 


at  out].  Here  5  is  a  nonlinear  function  that  helps  to  suppress  crosstalk  between  stored  pairs  by  forcing 
field  values  back  to  —1  or  ->-1  (it  is  described  below).  Computation  of  v  in  turn  yields 

4>'  =  S(^,  0''!') 

(47) 

at  outj.  On  the  next  iteration  we  get  iv' "  =  S{ip’,  and  =  S(4>’ ,  p'  'J'),  and  so  forth.  Each 

iteration  will  yield  a  closer  approximation  of  the  desired  (v'**,  ^^*^).  We  cannot  in  general  guarantee  that 
the  system  will  stabilize  (i.e.,  that  the  P  and  p  will  become  constant),  but  we  will  show  that  the  changes 
will  become  as  small  as  we  like.  Kosko  can  show  stability,  since  the  discreteness  of  his  fields  places  a  lower 
bound  on  nonzero  change. 

The  nonlinear  function  S  updates  the  output  field  in  the  following  way  [illustrated  for  the  computation 

w'  =  S{ip,  'iP)  j: 


P't  = 


+1  if 

Pi  if  -^2  ^  ^ 

-1  if  <  -On 


(48) 


Thus,  the  value  of  a  field  element  pi  is  not  changed  if  —On  ^  ^ i  •  p  ^  0^,  where  the  thresholds  0^,  0^  >  0. 
The  rule  for  p'  =  S{p,  p^)  is  analogous. 

Following  Hopfield^  we  show  that  the  stored  pairs  (,p^^\  ^tre  stable  states  in  the  dynamic  behavior 
of  the  memory. 

Theorem  (Stability  of  Stored  Pairs):  Any  stored  pair  is  stable,  provided  1 0]!  >  0,  where 
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#  max  ^2)-  (This  condition  holds  for  re&listic  values  of  the  thresholds.) 


m 


Ptvmf:  Suppose  that  is  ons  of  1*^0  stored  pairs  and  observe: 


=  S  A 


=  2  ■  d)W) 


The  expression  measures  the  correlation  between  <p^*'  and  which  varies  between  — |  fljl 


«d  “!  fijl  .  Notice  that  for  j^k  this  expression  has  mean  value  0.  On  the  other  hand,  when  j=k  its 


value  is  !  flu  .  Hence. 


Now  consider  the  crucial  expression  from  Eq.  48: 


as  tPj^')|nii 


If  tpl^^  =  +1,  then 


«  \n,\  >9, 


and  so  v  (  =  ~I-  Similarly,  if  »p|^^  =  -I,  then 


«  -Ifiii  <  -O2 


and  so  tp'/  =  -1.  In  both  cases  4>' ^  Since  tp'  =  and  (by  similar  reasoning)  0' 

see  that  any  stored  pair  <P^'^^)  is  stable.  ■ 


The  preceding  theorem  shows  that  the  stored  pairs  are  stable,  but  it  does  not  show  that  they  will  ever 


be  reached.  Therefore  we  prove  several  theorems  establishing  conditions  under  which  correct  recall  takes 


place.  The  following  theorem  shows  that  if  <p  is  sufficiently  close  to  one  0^^'^  and  sufficiently  far  from  all 


the  others,  then  perfect  recall  occurs  in  one  step. 


(Close  Matches  Lead  to  Correct  Recall):  0'  =  0^^',  provided  that 


(51) 


where  0  -  max  (®j,  Similarly,  <f>'  =  provided  -  0  >  ^ 

k*} 


Proof:  For  notational  convenience  let  be  the  similarity  of  and  <f>.  Thus  we  are  assuming 


<tj-b  >  x; 

(52) 


We  need  to  show  ij;'  =  so  consider  '^(p: 

=  (E  A 

* 

=  E  A 

k 

=  2  •  «^) 
it 

k 


(53) 


This  equation  is  plausible:  it  says  that  each  contributes  to  the  result  to  the  extent  the  corresponding 
^(*)  is  similar  to  <p.  From  Eq.  53  we  have 

k*j  (54) 


Now  we  consider  the  cases  =  +1  and  ~  ~1-  If  ~  +1  then 


-  <t>  =  +  E 

k*j 

^  -  E 

k^j 

>  e  ^  Bi 


Hence  ip/  =  +1  -  ipl’\  If  tp\^^  =  -1  then 


•  0  =  -<r^  +  E 

k-^i 

^  -ffj  +  E 

k*i 


-2' 


<  -0  ^  -$2 

Hence  0, '  =  —  1  =  In  either  case  The  proof  that  (j)'  =  is  analogous.  ■ 

The  preceding  proof  makes  very  pessimistic  assumptions,  namely,  that  the  other  are  negatively 
correlated  with  and  thus  maximally  interfering.  In  the  more  likely  case,  where  they  are  uncorrelated, 
or  even  better,  where  closeness  in  the  implies  closeness  in  the  we  can  get  correct  recall  in  spite  of 
the  other  being  similar  to  <j). 

Our  next  goal  is  to  show  that  the  system  converges  to  a  stable  state.  To  accomplish  this,  following 
Kosko®  and  Hopfield^,  we  investigate  how  a  Lyapunov  function  for  the  system  changes  in  time.  The  func¬ 


tion  is  defined: 


E[\p,  <p)  =  -Vs  —  0i)  -  Vs  -  ^2) 


(We  write  0^  ambiguously  for  d,T.)  This  formula  can  be  understood  by  observing  that  measures  the 

correlation  between  ip  and  '^(p,  and  ip-ip'^  measures  the  correlation  between  <p  and  Thus  E  represents 
the  “energy”  of  the  system,  which  decreases  as  these  correlations  increase.  Eq.  55  can  be  simplified  to: 

E{rp,  4>)  =  —xp^<p  -f  ip'0i  -f-  <p‘02 


where  we  write  ip^0  =  =  ip’9'<p.  We  now  use  this  energy  function  to  prove  some  important  results. 

Theorem  (Monotonicity  of  Change  of  Energy):  Changes  of  state  always  decrease  the  energy.  That 
is,  AE{ip)  <  0  and  AE{<p)  <  0. 


Proof:  The  change  in  energy  resulting  from  alteration  of  w  to  '  is: 


AE{ip)  =  E{r,4>)  -  E{rP,<P) 

=  {-xp'^ip  +  xp' -I-  <p'0-^  -  {^—tp^<P  -I-  ip'B]^  -f  <P'0-^ 

=  +  0)  •  -  9\) 

AE{ip)  =  -Alp  •  [^<p  -  0^) 


The  analysis  for  the  change  in  <p  is  analogous.  Expanding  the  inner  product  in  Eq.  57  yields: 


tf  "•  s*  i" 


'  rv  v  Ti*  -i'  *  v'l 


(58) 


A^(V’)  =  -  fl  ~  ^i) 

We  next  investigate  the  integrand. 

The  updating  equation  for  ^  (Ek).  48)  yields  the  following  possibilities: 

S  {0,  -^2}  if  >  0 

■  A0(  =0  if  -^2  $  ^^•(p  ^ 

Aw,  €  {0,  -2}  if  +  I?,  <  0 


Since  —  9-^  <  d,  <  0,  note  that  in  all  three  cases: 

Awt('tt-(p  -  9j)  ^  0 


I 

(60)  ! 


Furthermore,  note  that  if  Aw,  ^  0  then  this  inequality  is  strict, 
inequality  must  hold  for  a  set  of  t  with  nonsero  measure. 


Observe  that  since  Aw  ^  0.  this  strict 
This  implies  the  strict  monotonicity  of 


A£(V’)  “  -  jjj  -  ^i)  dt  <  0 


(61) 


Hence,  until  the  system  stabilizes,  every  4>  -*  xl>  step  of  the  iteration  decreases  the  energy.  Analogous  rea¬ 
soning  shows  that  the  energy  also  decreases  with  the  ib  ~~  <t>  steps.  ■ 

We  have  shown  that  every  step  decreases  the  energy.  We  now  show  that  the  energy  cannot  decrease 
forever,  because  it  has  a  lower  bound. 


Theorem  (Energy  Bounded  From  Below):  The  energy  is  bounded  from  below  by  a  quantity  that 
depends  only  on  't. 

Proof:  Recall  (Eq.  56)  that  the  energy  is  defined:  <p)  =  —rb'^tp  +  ^‘9^  -r  W'^2-  Since  4',  9^  and 

^2  ve  fixed,  there  are  clearly  bipolar  xb  &nd  4>  that  minimize  this  expression. 

We  can  derive  a  formula  for  a  explicit  lower  bound  as  follows.  Since  ^  1  and  ^  1,  we  have  the 
iscqaality: 
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H 


I  f  isj 


'.'C 

I'V’! 

i-f » 


I 

'  / 

I''!- 


i*,«  ■ 


/n,/n, 

$  r  r  A/  d«  dt 

=  MiHiiinji 

where  Af  =  max  Therefore, 

$t 

E{y},  d)  -  -rb^tp  -i-  yj-d^  4-  p-d^ 

^  — Afj  QjI  I  02]  f  <b‘02 

^  —  j\/|  Qjl  1 172!  ^ll  ^2l  ^2!  ^ll 

This  is  our  lower  bound.  It  is  fixed  since  it  depends  only  on  ■ 

The  preceding  theorems  show  chat  the  bidirectional  associative  field  memory  approaches  one  of  the 
states  characterizing  a  stored  associative  pair. 

5.  CONCLUSIONS 

We  have  argued  that  AI  is  moving  by  necessity  into  a  new  phase  that  recognises  the  role  of  nonproposi* 
tional  knowledge  in  intelligent  behavior.  We  also  argued  that  the  “new”  Al  must  make  use  of  massive 
parallelism  to  achieve  its  ends.  We  proposed  a  definition  of  massive  parallelism,  namely  that  the  number 
of  processing  elements  can  be  taken  as  a  continuous  quantity.  We  believe  that  this  definition  will 
encourage  the  development  of  the  necessary  theoretical  basis  for  neurocomputers,  optical  computers,  molec¬ 
ular  computers,  and  a  new  generation  of  analog  computers.  We  claimed  that  these  computing  technologies 
can  be  profitably  viewed  as  field  eomputeri,  computers  that  operate  on  entire  fields  of  data  in  parallel.  We 
discussed  the  importance  of  general  purpose  field  computers,  and  related  them  to  universal  field  computers. 
This  was  followed  by  a  theoretical  model  of  field  computation,  including  the- derivation  of  several  generali¬ 
zations  of  Taylor's  theorem  for  field  transformations.  These  theorems  provide  one  theoretical  basis  for 
universal  field  computers.  Finally,  we  illustrated  our  theory  of  field  computation  by  analyzing  a  continu¬ 
ous  field  version  of  Kosko’s  bidirectional  associative  memory 
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